Data and clock synchronization and variation compensation apparatus and method

ABSTRACT

An apparatus is provided for mitigating uncertainties in process, voltage, random, and systematic variations between first and second dies. The first die comprises a clock compensator to adjust one or more signal characteristics of an input clock, and to provide first and second clocks; a data transmitter to sample data with a version of the first clock and to transmit the sampled data to a data receiver of the second die, wherein the data receiver is to receive the sampled data and generate a received data; and a clock transmitter to transmit the second clock to a clock receiver of the second die, wherein the clock receiver is to generate a third clock, wherein a phase of the third clock is adjusted to generate a fourth clock, wherein a delayed version of the fourth clock is received by a sampler coupled to the data receiver to sample the received data.

CLAIM FOR PRIORITY

This application is a continuation of, and claims the benefit ofpriority to U.S. patent application Ser. No. 16/178,346, filed on Nov.1, 2018, titled “DATA AND CLOCK SYNCHRONIZATION AND VARIATIONCOMPENSATION APPARATUS AND METHOD,” and which is incorporated byreference in entirety.

BACKGROUND

To facilitate data transfer at high frequency (e.g., greater than 500Megabits per second (Mbps)) between integrated circuits (ICs) of silicondies in a package, while each die in the package may be on a differentprocess technology node and/or different operation power supply voltage,data sampling and clock synchronization between the dies can be verychallenging. Furthermore, on-die systematic and random variations withineach die, and also between dies, further exacerbate timing marginconstrains. Without proper mitigations, the on-die systematic and randomvariations within each die, and also between dies, introduces designcomplexity, degrades link data transfer rate and compromises the overallsystem performance, which leads to functional failures in dies.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments of the disclosure will be understood more fully from thedetailed description given below and from the accompanying drawings ofvarious embodiments of the disclosure, which, however, should not betaken to limit the disclosure to the specific embodiments, but are forexplanation and understanding only.

FIG. 1 illustrates a cross-section of a package with multiple dies andcircuitry for clock and data synchronization between dies and forcancelling effects of variations such as process, voltage, random andsystematic variations, according to some embodiments of the disclosure.

FIG. 2 illustrates a source synchronous architecture for programmablevariations compensation, in accordance with some embodiments.

FIG. 3 illustrates a timing diagram for the source synchronousprogrammable variations compensation scheme, in accordance with someembodiments.

FIG. 4 illustrates a flowchart of training clock alignment relative todata using the source synchronous programmable variations compensationscheme, in accordance with some embodiments.

FIG. 5 illustrates a common clock architecture for continuous anddynamic process, voltage, and temperature (PVT) variations compensation,in accordance with some embodiments.

FIG. 6 illustrates a flowchart of dynamic and continuous clock alignmentrelative to data using the apparatus of FIG. 5, in accordance with someembodiments.

FIG. 7 illustrates a smart device or a computer system or a SoC(System-on-Chip) having one or both of architectures of FIG. 2 and/orFIG. 5, according to some embodiments of the disclosure.

DETAILED DESCRIPTION

Ultra-high density packaging technology or die disaggregation technologyis a new technology and a process evolvement, enabling a significant diesize reduction (e.g., 4× reduction) which improves process yield, whileachieving high bandwidth and data rate (e.g., 1 Terabits per second(Tbps) bandwidth with 2.4 Gigabits per second (Gbps) data rate). Thisnew technology uses a high speed, yet simple and low power Single DataRate (SDR) die-to-die IO (input-output) link for multi-die communicationwithin a package, either side-by-side (e.g., using embedded multi dieinterconnect bridge—EMIB) or stacked (e.g., using interposer withthrough silicon via, TSV). The interposer may be an interposer die or anactive die having logic devices and transistors.

Die-to-die IO (input-output) architecture is a technique for diedisaggregation and multi-dimensional (e.g., 2.5D or 3D) multi-dieintegration. One example of die-to-die IO architecture utilizesdie-to-die source synchronous architecture. Without more, die-to-diesource synchronous architecture heavily relies on guard-bands duringtiming analysis to account for process, voltage, temperature, random andsystematic variations between dies. As such, with both dies being ondifferent process technology nodes and/or with different/separate supplyvoltages and/or temperature drifts, pre-silicon timing characterizationcan be very challenging, where too much guard-banding limits theconvergence frequency and increases design convergence effort, while toolittle guard-banding in timing analysis may cause speed path escapeesand functional failures.

Some embodiments describe a programmable low power die-to-die IOarchitecture that has the capability to operate robustly at highfrequencies (e.g., 3 GHz at 0.85 V), for source synchronousarchitecture, by cancelling uncertainties in process, voltage, randomand systematic variations between dice. In some embodiments, aprogrammable variation compensator is used with a training mechanism.The programmable low power die-to-die IO architecture comprises a sourceclock coarse compensator, a receiver clock fine compensator, a boot-uptraining scheme, and retraining mechanism. In some embodiments, thesource clock coarse compensator comprises a programmable clock delayline which is placed at the clock point of divergence (POD) to enablefine and/or coarse running for aligning two clock tree spines in thedifferent dies. The two clocks in the two clock tree spines may bealigned using software and/or hardware training. In some embodiments,the receiver clock fine compensator comprises a programmable pulse widthgenerator at the receiver to adjust the sampling clock edge in finesteps. This process is also referred to as fine clock tuning. In someembodiments, the boot-up training scheme applies an algorithm tooptimize timing margin by placing the sampling clock edge in the middleof the date eye thus cancelling out or mitigating (or reducing) process,voltage, and random and systematic variation across multiple dies. Insome embodiments, the retraining mechanism comprises hardware and/orsoftware to retraining cycles to track any VT (voltage and temperature)drifts in the sampling clock edge.

Some embodiments describe a continuous and dynamic process, voltage, andtemperature (PVT) and variation compensation apparatus. The apparatusprovides for dynamic tracking of any VT drifts while cancelling,mitigating or reducing out any static error (e.g., random and/orsystematic variations). In some embodiments, the apparatus comprises:dynamically programmable source clock compensator, continuous loopbackscheme, and decision making mechanism. In some embodiments, thedynamically programmable source clock compensator comprises aprogrammable clock delay line placed at the POD to enable fine and/orcoarse tuning for aligning the clocks of the two clock spines in thedifferent dies based on a feedback from a Finite State Machine (FSM). Insome embodiments, the continuous loopback scheme comprises circuitry forfeeding back the launching and sampling clocks in a matched manner, atthe end of the clock distribution, to a phase detector in a first diefor fine tuning and to provide dynamic threshold tracking. Any voltagedrift on the power supply or temperature shifts between the two dies canbe corrected with the continuous loopback scheme. In some embodiments,the decision making mechanism filters an output of the phase detectorand provides the filtered output to a finite state machine (FSM) whichadjusts the programmable delay line until both the sampling andgenerating clock edges are aligned.

There are many technical effects of the various embodiments. Forexample, the programmable low power die-to-die IO architecture and/orthe continuous and dynamic PVT and variation compensation apparatusallows for smaller clock skew penalty and ease design convergenceefforts thus providing significant frequency boost over prior schemes.The techniques of some embodiments also eliminates (or reduces) the needfor complex clock data recovery and the use of Delay Locked Loop (DLL)which are power hungry and consumes significant design effort. Theembodiments also reduce the latency in common clock architecture withoutthe use of a delay locked loop. The architectures of various embodimentsare modular, scalable due to digital technique and consumes lower power(e.g., 5× lower power) which fits into ultra-high density packaging ordie disaggregation applications. Other technical effects will be evidentfrom the various embodiments and figures.

In the following description, numerous details are discussed to providea more thorough explanation of embodiments of the present disclosure. Itwill be apparent, however, to one skilled in the art, that embodimentsof the present disclosure may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form, rather than in detail, in order to avoidobscuring embodiments of the present disclosure.

Note that in the corresponding drawings of the embodiments, signals arerepresented with lines. Some lines may be thicker, to indicate moreconstituent signal paths, and/or have arrows at one or more ends, toindicate primary information flow direction. Such indications are notintended to be limiting. Rather, the lines are used in connection withone or more exemplary embodiments to facilitate easier understanding ofa circuit or a logical unit. Any represented signal, as dictated bydesign needs or preferences, may actually comprise one or more signalsthat may travel in either direction and may be implemented with anysuitable type of signal scheme.

Throughout the specification, and in the claims, the term “connected”means a direct connection, such as electrical, mechanical, or magneticconnection between the things that are connected, without anyintermediary devices.

The term “coupled” means a direct or indirect connection, such as adirect electrical, mechanical, or magnetic connection between the thingsthat are connected or an indirect connection, through one or morepassive or active intermediary devices.

The term “adjacent” here generally refers to a position of a thing beingnext to (e.g., immediately next to or close to with one or more thingsbetween them) or adjoining another thing (e.g., abutting it).

The term “circuit” or “module” may refer to one or more passive and/oractive components that are arranged to cooperate with one another toprovide a desired function.

The term “signal” may refer to at least one current signal, voltagesignal, magnetic signal, or data/clock signal. The meaning of “a,” “an,”and “the” include plural references. The meaning of “in” includes “in”and “on.”

The term “scaling” generally refers to converting a design (schematicand layout) from one process technology to another process technologyand subsequently being reduced in layout area. The term “scaling”generally also refers to downsizing layout and devices within the sametechnology node. The term “scaling” may also refer to adjusting (e.g.,slowing down or speeding up—i.e. scaling down, or scaling uprespectively) of a signal frequency relative to another parameter, forexample, power supply level. The terms “substantially,” “close,”“approximately,” “near,” and “about,” generally refer to being within+/−10% of a target value.

Unless otherwise specified the use of the ordinal adjectives “first,”“second,” and “third,” etc., to describe a common object, merelyindicate that different instances of like objects are being referred toand are not intended to imply that the objects so described must be in agiven sequence, either temporally, spatially, in ranking or in any othermanner.

For the purposes of the present disclosure, phrases “A and/or B” and “Aor B” mean (A), (B), or (A and B). For the purposes of the presentdisclosure, the phrase “A, B, and/or C” means (A), (B), (C), (A and B),(A and C), (B and C), or (A, B and C).

The terms “left,” “right,” “front,” “back,” “top,” “bottom,” “over,”“under,” and the like in the description and in the claims, if any, areused for descriptive purposes and not necessarily for describingpermanent relative positions.

It is pointed out that those elements of the figures having the samereference numbers (or names) as the elements of any other figure canoperate or function in any manner similar to that described but are notlimited to such.

For purposes of the embodiments, the transistors in various circuits andlogic blocks described here are metal oxide semiconductor (MOS)transistors or their derivatives, where the MOS transistors includedrain, source, gate, and bulk terminals. The transistors and/or the MOStransistor derivatives also include Tri-Gate and FinFET transistors,Gate All Around Cylindrical Transistors, Tunneling FET (TFET), SquareWire, or Rectangular Ribbon Transistors, ferroelectric FET (FeFETs), orother devices implementing transistor functionality like carbonnanotubes or spintronic devices. MOSFET symmetrical source and drainterminals i.e., are identical terminals and are interchangeably usedhere. A TFET device, on the other hand, has asymmetric Source and Drainterminals. Those skilled in the art will appreciate that othertransistors, for example, Bi-polar junction transistors (BJT PNP/NPN),BiCMOS, CMOS, etc., may be used without departing from the scope of thedisclosure.

FIG. 1 illustrates a cross-section of a package 100 with multiple diesand circuitry for clock and data synchronization between dies and forcancelling effects of variations such as process, voltage, random andsystematic variations, according to some embodiments of the disclosure.Package 100 comprises package substrate 101, first die 102 (Die 1),second die 103 (Die 2), third die 104 (Die 3), and fourth die 105 (Die4). These dies can be of the same process technology node or differentprocess technology nodes.

For example, Die 1 and Die 2 may be on a 7 nm (nanometer) CMOS processtechnology node, Die 3 may be on a 10 nm CMOS process technology node,Die 4 may be a GaAs process technology node. For ease of reducing signallatency, the active regions of the dies may face one another when thedies are stacked vertically along the z-axis. Here, active region of die102 is illustrated as region 102 a, active region of die 103 isillustrated as region 103 a, active region of die 104 is illustrated asregion 104 a, and active region of die 105 is illustrated as region 105a.

In some embodiments, the dies are communicatively coupled to otheranother via through silicon vias (TSVs) that allow for communicationalong the z-axis. For example, TSVs (or any suitable interconnect or viatechnologies) 109 and 110 can be used to communicate between PCB and thedies along the z-direction. In some embodiments, a silicon interposer inthe redistribution layer (RDL) is used for communication between diesalong the x-axis. For example, silicon interposer 111 in RDL is used tocommunicate between Die 1 and Die 2. The package substrate 101 iscoupled to a printed circuit board (PCB) via bumps 106. One example, ofthe bumps 106 are solder balls. Similar solder balls 107 can be used tocouple the dies with package substrate 101. In some embodiments, microbumps are used to couple dies within a stack. For example, micro bumps108 are used to couple Die 3 and Die 4 with Die 1.

In various embodiments, programmable low power die-to-die IOarchitectures (e.g., 112, 113, 114, and/or 115) are provided that havethe capabilities to operate robustly at high frequencies (e.g., 3 GHzand higher at 0.85 V), for source synchronous architecture, bycancelling uncertainties in process, voltage, random and systematicvariations between dies. In some embodiments, low power die-to-die IOarchitectures (e.g., 112, 113, 114, and/or 115) are provided that havethe capabilities to operate robustly at high frequencies (e.g., 3 GHzand higher at 0.85 V), for common clock architecture, by dynamically andcontinuously cancelling uncertainties in process, voltage, random andsystematic variations between dies.

FIG. 2 illustrates a source synchronous architecture 200 forprogrammable variations compensation, in accordance with someembodiments. Architecture 200 comprises apparatus which is distributedbetween Die 1 (102) and Die 2 (103). Die 1 comprises logic 201 (e.g.,finite state machine (FSM)), a clock source 202 (e.g., oscillator, phaselocked loop (PLL), digital PLL, analog PLL, mixed-signal PLL, LC-PLL,etc.), transmitter portion of data lane(s), transmitter portion of clocklane(s), clock compensators 203 and 204, transmitter (Tx) clockdistribution network 205. Die 2 comprises receiver portion of datalane(s), receiver portion of clock lane(s), programmable pulse widthgenerator 217, and receiver (Rx) clock distribution network 218. Forsake of not obscuring the embodiments, simplified versions oftransmitters and receivers are shown.

The Data Lane(s) may include n-bits of data lanes, each having its owntransmission path from die 1 to die 2. In some embodiments, thetransmitter portion of the data lane comprises a synchronous transmittercircuitry 206 including a data sampler 207 coupled to an asynchronoustransmitter circuitry including pre-driver B1 210, electrostaticdischarge (ESD) diodes D1 and D2. The data to be transmitted is xx_datawhich propagates through the transmitter die to the receiver die viasilicon interposer RDL 212 (same as 111). In embodiments where dies arestacked, the silicon interposer RDL 212 may be replaced with TSVs suchas TSV 110 or with micro bumps (μBumps) such as μBumps 108. The receiverportion of data lane comprises a synchronous receiver circuitry 214including a data sampler 216 coupled to an asynchronous receivercircuitry including amplifier B2 220 and ESD diodes D1 and D2. Thetransmitter portion of the clock lane comprises an identical transmitteras the data transmitter, but without the sampler, for example.

In this example, transmitter portion of the clock lane comprisesasynchronous transmitter 209 having pre-driver B1 211 coupled to ESDdiodes D1 and D2. The clock to be transmitted is xx_clk which propagatesthrough the transmitter die to the receiver die via silicon interposerRDL 213 (same as 111). In embodiments where dies are stacked, thesilicon interposer RDL 213 may be replaced with TSVs such as TSV 110.The receiver portion of clock lane comprises an asynchronous receivercircuitry 215 including amplifier B2 221 and ESD diodes D1 and D2. Aperson skilled in the art would appreciate that implementations oftransmitters and receivers may include many other circuits such asimpedance terminations, clock-to-output (TCO) delay buffers, minimumdelay buffers, amplifiers, bias generators, etc., but are not shown toavoid obscuring the various embodiments.

In some embodiments, PLL 202 of Die 1 generates a high frequency clock,which is sent to two identical clock compensators 203 and 204. Theseclock compensators are placed at the point of divergence (POD) of thedata and clock lanes. In some embodiments, compensators 203 and 204 havethe ability to tune the launching and sampling clocks independentlythrough coarse and/or fine settings. These setting may be provided bylogic 201. In some embodiments, during boot-up prior to any training, adefault safe fuse value is loaded into the compensators 203 and 204 toset or initialize the delay configuration and/or other performanceparameters (e.g., slew rate, pulse width, etc.). The launch clock isthen distributed via recombinant clock tree 205 to ensure clock skewbetween data lanes are kept at minimal. In each data lane, asynchronizer 207 is used to re-time transmit data on a micro-bump viadriver 210 and RDL 212.

On the other hand, sampling clock 208 is sent through an asynchronoustransmitter (ASYNCTX) block 209 which is replica version of SynchronousTransmitter (SYNCTX) 206. The synchronized data lanes and clock are sentsource synchronously to the Die 2 103 through pre-driver and driver inB1 210 and 211, respectively. To keep the architecture simple and lowpower, the pre-driver and driver are constructed with two stages ofinverters sending a full swing CMOS signaling through the siliconinterposer 212, 213 to Die 2 receivers accordingly—Synchronous Receiver(SYNCRX) 214 and Asynchronous Receiver (ASYNCRX) 215.

Die 2's SYNCRX 214 and ASYNCRX 215 circuitries are exactly the same, butSYNCRX has a latch 216 as a sampler while ASYNCRX goes through aprogrammable pulse width generator 217 and a clock tree distribution 218that matches to Die 1's clock tree distribution 205 design in terms ofbuffering stages and RC (resistance-capacitance propagation delay). Insome embodiments, the programmable pulse width generator 217 has thecapability to shape the clock's pulse width and finely tune the samplingclock edges through a boot-up training mechanism.

A boot-up, training mechanism uses software algorithms and/or hardwarelogic to tune the performance parameters of clock compensators 203 and204, and programmable pulse width generator 217 to reliably cancel outprocess, voltage, systematic and random variations between two dies.During boot-up, each data lane sends a continuous test pattern (e.g.,pattern 1010 . . . ) from Die 1 to Die 2, where an optimization flowsweeps through all possible settings of programmable pulse widthgenerator 217 and picks the settings that provide the best overallperformance as measured by the eye width at the sampler 216. Thesoftware algorithm and/or hardware logic then places the sampling edgein the middle of the widest passing window of the data eye. Due to noiseconcerns, a majority vote counter may be used to ensure adequate noisesuppression. In some cases, it is possible for the algorithm or hardwarelogic to not find any passing window due to huge mismatch in the clockdistributions, 205 and 218 between the dies due to process nodes orvoltage drifts. In one such case, the clock compensators 203, 204 can becoarsely trained by delaying the clock lanes or the data lanesindependently, such that the clock edge from the pulse width generator217 falls within the data pulses.

FIG. 3 illustrates a timing diagram 300 for the source synchronousprogrammable variations compensation scheme, in accordance with someembodiments.

FIG. 4 illustrates a flowchart 400 of training clock alignment relativeto data using the source synchronous programmable variationscompensation scheme, in accordance with some embodiments. The variousoperation block shown in flowchart can be performed out of order, inparallel, or in the order shown. For example, blocks 401 and 402 can beperformed in parallel or simultaneously.

At block 401, the delay and/or other performance parameters of clockcompensators 203 and 204 are set or initialized. For example, uponboot-up, a fuse value(s) is read are provided to clock compensators 203and 204 set their drive strengths, propagation delay, and/or slew rate,etc. At block 402, the pulse width generator 217 is also initialized orset. For example, the delay of a falling edge of a clock (wherein thedelay is an example of a performance parameter of the pulse widthgenerator 217) is set or initialized. In some embodiments, fuse valuesat boot-up or a training phase are read and used to initialize theperformance parameters of clock compensators 203 and 204 and pulse widthgenerator 217. In some embodiments, the operating system or any softwarecan also be used to initialize the performance parameters of clockcompensators 203 and 204 and pulse width generator 217.

At block 403, results of sampling received data using the received clockis analyzed to determine whether correct data was captured. Thereafter,at block 404, setting for the pulse width generator 217 is swept fromsmallest delay to larger delay in a fine step (e.g., 1 picosecond (ps)step). For instance, the falling edge of the received clock is pushedout by 1 ps step and results of sampling received data using thereceived clock is analyzed to determine whether correct data wascaptured. Once the process of sweeping or pushing out (or alternativelypulling in) the falling clock edge is complete, the process proceeds toblock 405. At block 405, a determination is made after the passing timewindow in which data was correctly sampled for a given delayed fallingedge clock. A boot-up, training mechanism uses software algorithmsand/or hardware logic to tune the performance parameters of clockcompensators 203 and 204, and programmable pulse width generator 217 toreliably cancel out process, voltage, systematic and random variationsbetween two dies. During boot-up, each data lane sends a continuous testpattern (e.g., pattern 1010 . . . ) from Die 1 to Die 2, where anoptimization flow sweeps through all possible settings of programmablepulse width generator 217 and picks the settings that provide the bestoverall performance as measured by the eye width at the sampler 216.This setting for the best overall performance indicates the passingwindow.

Once the passing window is determined (e.g., once it is figured outwhere the falling edge of the received clock should be placed forcorrectly sampling the data), the process proceeds to block 406. Atblock 406, the performance parameters of clock compensators 203 and 204,and programmable pulse width generator 217 are set to achieve the bestoverall performance. For example, the performance parameters of clockcompensators 203 and 204, and programmable pulse width generator 217 areset to cause the received falling edge of the clock to be centered in adata eye to sample data correctly with most timing margin.

FIG. 5 illustrates a common clock architecture 500 for continuous anddynamic process, voltage, and temperature (PVT) variations compensation,in accordance with some embodiments. Architecture 500 is almost similarto the architecture 200 but has added 3 additional micro bumps toprovide continuous feedback without the need for dead cycles forretraining. Here, the programmable pulse width generator is alsoremoved. However, some embodiments can have the programmable pulse widthgenerator that operates same as programmable pulse width generator 217.

Architecture 500 comprises an apparatus which is distributed between Die1 (102) and Die 2 (103). Die 1 comprises a clock source 301 (e.g.,oscillator, phase locked loop (PLL), digital PLL, analog PLL,mixed-signal PLL, LC-PLL, etc.), transmitter portion of data lane(s),transmitter portion of clock lane(s), clock compensators 302 and 303,transmitter (Tx) clock distribution network 304, phase detector 329, FSM332, etc. Die 2 comprises receiver portion of data lane(s), receiverportion of clock lane(s), receiver (Rx) clock distribution network 317.For sake of not obscuring the embodiments, simplified versions oftransmitters and receivers are shown.

The Data Lane(s) may include n-bits of data lanes, each having its owntransmission path from die 1 to die 2. In some embodiments, thetransmitter portion of the data lane comprises a synchronous transmittercircuitry 305 including a data sampler 306 coupled to an asynchronoustransmitter circuitry including pre-driver B1 308, electrostaticdischarge (ESD) diodes D1 and D2. The data to be transmitted is xx_datawhich propagates through the transmitter die to the receiver die viasilicon interposer RDL 310 (same as 111). In embodiments where dies arestacked, the silicon interposer RDL 310 may be replaced with TSVs suchas TSV 110 or with micro bumps (μBumps) such as μBumps 108. The receiverportion of data lane comprises a synchronous receiver circuitry 312including a data sampler 316 coupled to an asynchronous receivercircuitry including amplifier B2 314 and ESD diodes D1 and D2. Thetransmitter portion of the clock lane comprises an identical transmitteras the data transmitter, but without the sampler, for example.

In this example, transmitter portion of the clock lane comprisesasynchronous transmitter 307 having pre-driver B1 309 coupled to ESDdiodes D1 and D2. The clock to be transmitted is xx_clk which propagatesthrough the transmitter die to the receiver die via silicon interposerRDL 311 (same as 111). In embodiments where dies are stacked, thesilicon interposer RDL 311 may be replaced with TSVs such as TSV 110 orwith micro bumps (μBumps) such as μBumps 108. The receiver portion ofclock lane comprises an asynchronous receiver circuitry 313 includingamplifier B2 315 and ESD diodes D1 and D2. A person skilled in the artwould appreciate that implementations of transmitters and receivers mayinclude many other circuits such as impedance terminations,clock-to-output (TCO) delay buffers, minimum delay buffers, amplifiers,bias generators, etc.

In some embodiments, PLL 301 of Die 1 generates a high frequency clock,which is sent to two identical clock compensators 302 and 303. Theseclock compensators are placed at the POD of the data and clock lanes. Insome embodiments where it is desirable to align the outputs of thetransmitter (Tx) clock distribution 304 and the receiver (Rx) clockdistribution 317 in time, an earlier clock is sent to Die2 to mitigatethe additional delay through ASYNCTX 307, RDL311 and ASYNCRX 313 suchthat total delay from POD to Tx Clock distribution 304 and to Rx ClockDistribution 317 are nominally matched. In some embodiments,compensators 302 and 303 have the ability to tune the launching andsampling clocks independently through coarse and/or fine settings. Thesesetting may be provided by logic FSM 332. In some embodiments, duringboot-up prior to any training, a default safe fuse value is loaded intothe compensators 302 and 303 to set or initialize the delayconfiguration and/or other performance parameters (e.g., slew rate,pulse width, etc.). The launch clock is then distributed via recombinantclock tree 304 to ensure clock skew between data lanes are kept atminimal. In each data lane, a synchronizer 306 is used to re-timetransmit data on a micro-bump via driver 308 and RDL 310.

On the other hand, sampling clock is sent through an asynchronoustransmitter (ASYNCTX) block 307 which is replica version of ASYNCTXblock of the data lane. The synchronized data lanes and clock are sentsource synchronously to the Die 2 103 through pre-driver and driver inB1 308 and 309, respectively. To keep the architecture simple and lowpower, the pre-driver and driver are constructed with two stages ofinverters sending a full swing CMOS signaling through the siliconinterposer 310, 311 to Die 2 receivers accordingly—Synchronous Receiver(SYNCRX) 312 and Asynchronous Receiver (ASYNCRX) 313.

Die 2's SYNCRX 312 and ASYNCRX 313 circuitries are exactly the same, butSYNCRX has a latch 316 as a sampler while ASYNCRX goes through a clocktree distribution 317 that matches to Die 1's clock tree distribution304 design in terms of buffering stages and RC (resistance-capacitancepropagation delay).

Here, a clock loopback path is provided from transmitter die 1 102 toreceiver die 2 103 and back to transmitter die 1, indicated by thedotted line. The phase detector 329 compares the clocks through theloopback path and the clock 318 from the clock distribution 317 at thereceiver die 2 to determine the uncertainties in the clocks. Theloopback path sends the transmit clock 322 which is an output of theclock distribution network 304. The transmit clock 322 is the same clockwhich is used by samplers 306 in the data lanes. The transmit clock isreceived by transmitter ASYNCTX 323 and send to receiver ASYNCRX 325 indie 2 via silicon (Si) interposer RDL 324. At the receiver, the clock isreceived and resent back to die 1 102 by transmitter 326. For example,transmitter ASYNCTX 326 in die 2 103 sends the received clock back todie 1 102 via Si Interposer RDL 327. The receiver ASYNCRX 328 at die 1102 receives the clock and generates the loopback clock 330, which isprovided as input B to phase detector 329. The other clock received bythe phase detector 329 is the long path receive clock which includes thepath 318, ASYNCTX 319, Si Interposer RDL 320, and ASYNCRX 321 as shown.

To capture the delay variations between data and clock I/O pathsdynamically, the phase difference between the clock distributions 304and 317 from die 1 and die 2 are sampled continuously with a phasedetector 329. In doing so, an identical loopback path for the launchingclock (point A to point B) is created to match the loopback path delayfor the sampling clock (point C to point D). Meanwhile, a single clockbit 322 and 318 is tapped out from each clock distribution network 304and 317 as part of the loopback paths. As a result, with both loopbackpaths identical, the phase difference detected is caused by themismatches in the clock distribution networks 304 and 317. In someembodiments where it is desirable to align the outputs of Tx clockdistribution 304 and Rx clock distribution 317 in time (e.g., clock bits322 and 318), the loopback path is shortened to bypass the Si InterposerRDL 324, ASYNCRX 325, ASYNCTX 326, Si Interposer RDL 327 by directlyconnecting TX_DIE_TM* out of ASYNCTX 323 to RX_Die_TM* of ASYNCRX 328.

Once the phase detector 329 detects a phase difference, a feedbacksignal is sent to the FSM 332. If the launching clock lags behind thesampling clock, the sampling clock is pushed out and vice versa. In someembodiments, the FSM 331 has built-in digital filtering to ignore clockskew or jitter, and aligns both clock edges and dynamically adjusts thesource clock compensators 302 delay through configuration settings 333.To prevent any functional failures during calibration, the FSM 332 mayallow a linear search with limited range when enabled while using deeperdigital filtering to average out any noise sources.

FIG. 6 illustrates a flowchart 600 of dynamic and continuous clockalignment relative to data using the apparatus of FIG. 5, in accordancewith some embodiments. The various operation block shown in flowchartcan be performed out of order, in parallel, or in the order shown. Forexample, blocks 602 and 603 can be performed in parallel orsimultaneously.

At block 601, the delay and/or other performance parameters of clockcompensators 302 and 303 are set or initialized. For example, uponboot-up, a fuse value(s) is read are provided to clock compensators 302and 303 set their drive strengths, propagation delay, and/or slew rate,etc. In some embodiments, FSM 332 provides initial setting or values forthe performance parameters of clock compensators 203 and 204.

At block 602, loopback clock is sent through the loopback path from die1 102 to die 2 103, and back to die 1 102. The loopback clock isreceived at point B at the input of phase detector 329. At block 603,receive clock 318 from the long path, which includes path from point Cthrough clock distribution 317, through Si interposer 320, is receivedat point D at the input phase detector 329.

At block 604, phase detector 329 compares the clocks at points B and Dand FSM 332 determines from the output of phase detector 329 whether theloopback clock and the long path clock have aligned phases. If thephases are not aligned, then FSM 332 adjusts one or more performanceparameters of clock compensators 302 and 303. For example, propagationdelay of the clock compensators either 302 or 303 is increased ordecreased at block 605. The process then proceeds back to block 602. Ifthe phases are aligned, then FSM 332 maintains the one or moreperformance parameters of clock compensators 302 and 303. For example,propagation delay of the clock compensators 302 and 303 is maintained atblock 606. This process may be applied continuously for dynamicallychanging the one or more performance parameters of clock compensators302 and 303 to align the loopback clock and the long path clock, asindicated by the dotted line.

Architectures 200 and 500 enable die-to-die IO timing trainings andoptimization between data and clock across two dies with minimaloverhead in terms power and performance, that make the system robust andhigh speed (e.g., data rate as high as 3 Gbps). The second architecture500 offers a continuous tracking scheme to nullify the effects oftemperature and voltage drifts in the system. As a result, thesearchitectures may not need a complex controller and physical (PHY) layerdesign, and hence broaden the I/O link usage for most of the diedisaggregation applications.

In some embodiments, pulse width generator 217 can also be added toarchitecture 500 to allow for an additional knob to adjust the clockedge. The pulse width generator 217 in that case can be controlled byany suitable means. For example, pulse width generator 217 can becontrolled by FSM 332 and/or by software (e.g., operating system).

In some embodiments, the loopback clock path and the long clock pathalong with the phase detector and FSM 332 can also be added toarchitecture 200 to provide additional knobs for centering the samplingclock edge in the middle of the data eye for the source synchronousarchitecture. The dynamic and continuous compensation of the process,voltage, random and systematic variations can be applied to botharchitectures 200 and 500, in accordance with some embodiments.

FIG. 7 illustrates a smart device or a computer system or a SoC(System-on-Chip) having one or both of architectures of FIG. 2 and/orFIG. 5, according to some embodiments of the disclosure. FIG. 7illustrates a block diagram of an embodiment of a mobile device in whichflat surface interface connectors could be used. In some embodiments,computing device 1600 represents a mobile computing device, such as acomputing tablet, a mobile phone or smart-phone, a wireless-enablede-reader, or other wireless mobile device. It will be understood thatcertain components are shown generally, and not all components of such adevice are shown in computing device 1600.

In some embodiments, computing device 1600 includes first processor 1610(e.g., die 1 102) coupled to second processor 1690 (e.g., die 2 103)having an apparatus of FIG. 2 or FIG. 5 distributed between the two diesaccording to some embodiments discussed. Other blocks of the computingdevice 1600 may also include an apparatus of FIG. 2 or FIG. 5 forcancelling PVT and random/systematic variations between and across dies,according to some embodiments. The various embodiments of the presentdisclosure may also comprise a network interface within 1670 such as awireless interface so that a system embodiment may be incorporated intoa wireless device, for example, cell phone or personal digitalassistant. In some embodiments, precessor1 1610 itself may be comprisedof many dies connected through μBumps, and RDLs as shown in FIG. 1.

Referring back to FIG. 7, in some embodiments, processor 1610 (and/orprocessor 1690) can include one or more physical devices, such asmicroprocessors, application processors, microcontrollers, programmablelogic devices, or other processing means. The processing operationsperformed by processor 1610 include the execution of an operatingplatform or operating system on which applications and/or devicefunctions are executed. The processing operations include operationsrelated to I/O (input/output) with a human user or with other devices,operations related to power management, and/or operations related toconnecting the computing device 1600 to another device. The processingoperations may also include operations related to audio I/O and/ordisplay I/O.

In some embodiments, computing device 1600 includes audio subsystem1620, which represents hardware (e.g., audio hardware and audiocircuits) and software (e.g., drivers, codecs) components associatedwith providing audio functions to the computing device. Audio functionscan include speaker and/or headphone output, as well as microphoneinput. Devices for such functions can be integrated into computingdevice 1600 or connected to the computing device 1600. In oneembodiment, a user interacts with the computing device 1600 by providingaudio commands that are received and processed by processor 1610.

In some embodiments, computing device 1600 comprises display subsystem1630. Display subsystem 1630 represents hardware (e.g., display devices)and software (e.g., drivers) components that provide a visual and/ortactile display for a user to interact with the computing device 1600.Display subsystem 1630 includes display interface 1632, which includesthe particular screen or hardware device used to provide a display to auser. In one embodiment, display interface 1632 includes logic separatefrom processor 1610 to perform at least some processing related to thedisplay. In one embodiment, display subsystem 1630 includes a touchscreen (or touch pad) device that provides both output and input to auser.

In some embodiments, computing device 1600 comprises I/O controller1640. I/O controller 1640 represents hardware devices and softwarecomponents related to interaction with a user. I/O controller 1640 isoperable to manage hardware that is part of audio subsystem 1620 and/ordisplay subsystem 1630. Additionally, I/O controller 1640 illustrates aconnection point for additional devices that connect to computing device1600 through which a user might interact with the system. For example,devices that can be attached to the computing device 1600 might includemicrophone devices, speaker or stereo systems, video systems or otherdisplay devices, keyboard or keypad devices, or other I/O devices foruse with specific applications such as card readers or other devices.

As mentioned above, I/O controller 1640 can interact with audiosubsystem 1620 and/or display subsystem 1630. For example, input througha microphone or other audio device can provide input or commands for oneor more applications or functions of the computing device 1600.Additionally, audio output can be provided instead of or in addition todisplay output. In another example, if display subsystem 1630 includes atouch screen, the display device also acts as an input device, which canbe at least partially managed by I/O controller 1640. There can also beadditional buttons or switches on the computing device 1600 to provideI/O functions managed by I/O controller 1640.

In some embodiments, I/O controller 1640 manages devices such asaccelerometers, cameras, light sensors or other environmental sensors,or other hardware that can be included in the computing device 1600. Theinput can be part of direct user interaction, as well as providingenvironmental input to the system to influence its operations (such asfiltering for noise, adjusting displays for brightness detection,applying a flash for a camera, or other features).

In some embodiments, computing device 1600 includes power management1650 that manages battery power usage, charging of the battery, andfeatures related to power saving operation. Memory subsystem 1660includes memory devices for storing information in computing device1600. Memory can include nonvolatile (state does not change if power tothe memory device is interrupted) and/or volatile (state isindeterminate if power to the memory device is interrupted) memorydevices. Memory subsystem 1660 can store application data, user data,music, photos, documents, or other data, as well as system data (whetherlong-term or temporary) related to the execution of the applications andfunctions of the computing device 1600.

Elements of embodiments are also provided as a machine-readable medium(e.g., memory 1660) for storing the computer-executable instructions(e.g., instructions to implement any other processes discussed herein).The machine-readable medium (e.g., memory 1660) may include, but is notlimited to, flash memory, optical disks, CD-ROMs, DVD ROMs, RAMs,EPROMs, EEPROMs, magnetic or optical cards, phase change memory (PCM),or other types of machine-readable media suitable for storing electronicor computer-executable instructions. For example, embodiments of thedisclosure may be downloaded as a computer program (e.g., BIOS) whichmay be transferred from a remote computer (e.g., a server) to arequesting computer (e.g., a client) by way of data signals via acommunication link (e.g., a modem or network connection).

In some embodiments, computing device 1600 comprises connectivity 1670.Connectivity 1670 includes hardware devices (e.g., wireless and/or wiredconnectors and communication hardware) and software components (e.g.,drivers, protocol stacks) to enable the computing device 1600 tocommunicate with external devices. The computing device 1600 could beseparate devices, such as other computing devices, wireless accesspoints or base stations, as well as peripherals such as headsets,printers, or other devices.

Connectivity 1670 can include multiple different types of connectivity.To generalize, the computing device 1600 is illustrated with cellularconnectivity 1672 and wireless connectivity 1674. Cellular connectivity1672 refers generally to cellular network connectivity provided bywireless carriers, such as provided via GSM (global system for mobilecommunications) or variations or derivatives, CDMA (code divisionmultiple access) or variations or derivatives, TDM (time divisionmultiplexing) or variations or derivatives, or other cellular servicestandards. Wireless connectivity (or wireless interface) 1674 refers towireless connectivity that is not cellular and can include personal areanetworks (such as Bluetooth, Near Field, etc.), local area networks(such as Wi-Fi), and/or wide area networks (such as WiMax), or otherwireless communication.

In some embodiments, computing device 1600 comprises peripheralconnections 1680. Peripheral connections 1680 include hardwareinterfaces and connectors, as well as software components (e.g.,drivers, protocol stacks) to make peripheral connections. It will beunderstood that the computing device 1600 could both be a peripheraldevice (“to” 1682) to other computing devices, as well as haveperipheral devices (“from” 1684) connected to it. The computing device1600 commonly has a “docking” connector to connect to other computingdevices for purposes such as managing (e.g., downloading and/oruploading, changing, synchronizing) content on computing device 1600.Additionally, a docking connector can allow computing device 1600 toconnect to certain peripherals that allow the computing device 1600 tocontrol content output, for example, to audiovisual or other systems.

In addition to a proprietary docking connector or other proprietaryconnection hardware, the computing device 1600 can make peripheralconnections 1680 via common or standards-based connectors. Common typescan include a Universal Serial Bus (USB) connector (which can includeany of a number of different hardware interfaces), DisplayPort includingMiniDisplayPort (MDP), High Definition Multimedia Interface (HDMI),Firewire, or other types.

Reference in the specification to “an embodiment,” “one embodiment,”“some embodiments,” or “other embodiments” means that a particularfeature, structure, or characteristic described in connection with theembodiments is included in at least some embodiments, but notnecessarily all embodiments. The various appearances of “an embodiment,”“one embodiment,” or “some embodiments” are not necessarily allreferring to the same embodiments. If the specification states acomponent, feature, structure, or characteristic “may,” “might,” or“could” be included, that particular component, feature, structure, orcharacteristic is not required to be included. If the specification orclaim refers to “a” or “an” element, that does not mean there is onlyone of the elements. If the specification or claims refer to “anadditional” element, that does not preclude there being more than one ofthe additional element.

Furthermore, the particular features, structures, functions, orcharacteristics may be combined in any suitable manner in one or moreembodiments. For example, a first embodiment may be combined with asecond embodiment anywhere the particular features, structures,functions, or characteristics associated with the two embodiments arenot mutually exclusive.

While the disclosure has been described in conjunction with specificembodiments thereof, many alternatives, modifications and variations ofsuch embodiments will be apparent to those of ordinary skill in the artin light of the foregoing description. The embodiments of the disclosureare intended to embrace all such alternatives, modifications, andvariations as to fall within the broad scope of the appended claims.

In addition, well known power/ground connections to integrated circuit(IC) chips and other components may or may not be shown within thepresented figures, for simplicity of illustration and discussion, and soas not to obscure the disclosure. Further, arrangements may be shown inblock diagram form in order to avoid obscuring the disclosure, and alsoin view of the fact that specifics with respect to implementation ofsuch block diagram arrangements are highly dependent upon the platformwithin which the present disclosure is to be implemented (i.e., suchspecifics should be well within purview of one skilled in the art).Where specific details (e.g., circuits) are set forth in order todescribe example embodiments of the disclosure, it should be apparent toone skilled in the art that the disclosure can be practiced without, orwith variation of, these specific details. The description is thus to beregarded as illustrative instead of limiting.

An abstract is provided that will allow the reader to ascertain thenature and gist of the technical disclosure. The abstract is submittedwith the understanding that it will not be used to limit the scope ormeaning of the claims. The following claims are hereby incorporated intothe detailed description, with each claim standing on its own as aseparate embodiment.

What is claimed is:
 1. An apparatus comprising: a first adjustablebuffer to receive an input clock and provide a first clock; a secondadjustable buffer to receive the input clock and provide a second clock,wherein the first and second adjustable buffers are to modify one ormore signal characteristics of the first and the second clocks,respectively, according to one or more controls; a first transmitter tosample data with a delayed version of the first clock, and to transmitthe sampled data to a first receiver; and a second transmitter totransmit the second clock to a second receiver, wherein the firstreceiver and the second receiver are in a different die than a diecomprising the first transmitter and the second transmitter.
 2. Theapparatus of claim 1, wherein the first receiver is to sample a receivedversion of the sampled data with a third clock.
 3. The apparatus ofclaim 2, wherein the second receiver is to generate a fourth clock whichis provided as input to a clock distribution network to generate a fifthclock, and wherein the third clock is a delayed version of the fifthclock.
 4. The apparatus of claim 3 further comprising a phase detectorto compare a version of the third clock with a loop-back version of thedelayed version of the first clock, and to provide a comparison result.5. The apparatus of claim 4 further comprising logic to adjust a logicvalue of the one or more control signals according to the comparisonresult.
 6. The apparatus of claim 4, wherein the loop-back version, ofthe delayed version of the first clock, is generated within a same die.7. The apparatus of claim 4, wherein the loop-back version, of thedelayed version of the first clock, is generated by a transmission pathbetween two separate dies.
 8. The apparatus of claim 5, wherein thefirst receiver and the second receiver are in a first die while thefirst and second adjustable buffers, the first and second transmitters,the phase detector and the logic are in a second die.
 9. The apparatusof claim 5 comprises a filter to filter the comparison result andprovide a filtered version of the comparison result to the logic. 10.The apparatus of claim 8, wherein the first and second dies are coupledvia a silicon interposer or a through silicon via.
 11. The apparatus ofclaim 8, wherein the first and second dies are positioned in a packagealong a same plane.
 12. The apparatus of claim 8, wherein the first andsecond dies are positioned in a package in different planes.
 13. Anapparatus comprising: a first circuitry to adjust one or more signalcharacteristics of an input clock, and to provide first and secondclocks; a first transmitter to sample data, for transmission, with aversion of the first clock and to transmit the sampled data to a firstreceiver; and a second transmitter to transmit the second clock to asecond receiver, wherein the first receiver and the second receiver arein a different die than a die comprising the first transmitter and thesecond transmitter.
 14. The apparatus of claim 13, wherein the datareceiver is to receive the sampled data and to generate a received data.15. The apparatus of claim 14, wherein the second receiver is to receivethe second clock and generate a third clock, wherein a phase of thethird clock is adjusted by a pulse width generator to generate a fourthclock.
 16. The apparatus of claim 15, wherein a delayed version of thefourth clock is received by a sampler coupled to the first receiver tosample the received data.
 17. The apparatus of claim 13, wherein thefirst and second dies are coupled to one another via a siliconinterposer or a through-silicon-via (TSV).
 18. A system-on-chipcomprising: a memory; a first die coupled to the memory; a second diecoupled to the first die; and a communication interface to allow thefirst and/or second dies to communicate with another device, wherein thefirst die includes: a first adjustable buffer to receive an input clockand provide a first clock; a second adjustable buffer to receive theinput clock and provide a second clock, wherein the first and secondadjustable buffers are to modify one or more signal characteristics ofthe first and the second clocks, respectively, according to one or morecontrols; a first transmitter to sample data with a delayed version ofthe first clock, and to transmit the sampled data to a first receiver;and a second transmitter to transmit the second clock to a secondreceiver.
 19. The system-on-chip of claim 18, wherein the first receiveris to sample a received version of the sampled data with a third clock.20. The system-on-chip of claim 19, wherein the second receiver is togenerate a fourth clock which is provided as input to a clockdistribution network to generate a fifth clock, and wherein third clockis a delayed version of the fifth clock.
 21. The system-on-chip of claim20 further comprising a phase detector to compare a version of the thirdclock with a loop back version of the delayed version of the firstclock, and to provide a comparison result.
 22. The system-on-chip ofclaim 21 further comprising logic to adjust a logic value of the one ormore control signals according to the comparison result.